By JONAH LEHRER

Published: February 25, 2010

The Victorians had many names for depression, and Charles Darwin used them all. There were his “fits” brought on by “excitements,” “flurries” leading to an “uncomfortable palpitation of the heart” and “air fatigues” that triggered his “head symptoms.” In one particularly pitiful letter, written to a specialist in “psychological medicine,” he confessed to “extreme spasmodic daily and nightly flatulence” and “hysterical crying” whenever Emma, his devoted wife, left him alone.

While there has been endless speculation about Darwin’s mysterious ailment — his symptoms have been attributed to everything from lactose intolerance to Chagas disease — Darwin himself was most troubled by his recurring mental problems. His depression left him “not able to do anything one day out of three,” choking on his “bitter mortification.” He despaired of the weakness of mind that ran in his family. “The ‘race is for the strong,’ ” Darwin wrote. “I shall probably do little more but be content to admire the strides others made in Science.”

Darwin, of course, was wrong; his recurring fits didn’t prevent him from succeeding in science. Instead, the pain may actually have accelerated the pace of his research, allowing him to withdraw from the world and concentrate entirely on his work. His letters are filled with references to the salvation of study, which allowed him to temporarily escape his gloomy moods. “Work is the only thing which makes life endurable to me,” Darwin wrote and later remarked that it was his “sole enjoyment in life.”

For Darwin, depression was a clarifying force, focusing the mind on its most essential problems. In his autobiography, he speculated on the purpose of such misery; his evolutionary theory was shadowed by his own life story. “Pain or suffering of any kind,” he wrote, “if long continued, causes depression and lessens the power of action, yet it is well adapted to make a creature guard itself against any great or sudden evil.” And so sorrow was explained away, because pleasure was not enough. Sometimes, Darwin wrote, it is the sadness that informs as it “leads an animal to pursue that course of action which is most beneficial.” The darkness was a kind of light.

The mystery of depression is not that it exists — the mind, like the flesh, is prone to malfunction. Instead, the paradox of depression has long been its prevalence. While most mental illnesses are extremely rare — schizophrenia, for example, is seen in less than 1 percent of the population — depression is everywhere, as inescapable as the common cold. Every year, approximately 7 percent of us will be afflicted to some degree by the awful mental state that William Styron described as a “gray drizzle of horror . . . a storm of murk.” Obsessed with our pain, we will retreat from everything. We will stop eating, unless we start eating too much. Sex will lose its appeal; sleep will become a frustrating pursuit. We will always be tired, even though we will do less and less. We will think a lot about death.

The persistence of this affliction — and the fact that it seemed to be heritable — posed a serious challenge to Darwin’s new evolutionary theory. If depression was a disorder, then evolution had made a tragic mistake, allowing an illness that impedes reproduction — it leads people to stop having sex and consider suicide — to spread throughout the population. For some unknown reason, the modern human mind is tilted toward sadness and, as we’ve now come to think, needs drugs to rescue itself.

The alternative, of course, is that depression has a secret purpose and our medical interventions are making a bad situation even worse. Like a fever that helps the immune system fight off infection — increased body temperature sends white blood cells into overdrive — depression might be an unpleasant yet adaptive response to affliction. Maybe Darwin was right. We suffer — we suffer terribly — but we don’t suffer in vain.

ANDY THOMSON IS a psychiatrist at the University of Virginia. He has a scruffy gray beard and steep cheekbones. When Thomson talks, he tends to close his eyes, as if he needs to concentrate on what he’s saying. But mostly what he does is listen: For the last 32 years, Thomson has been tending to his private practice in Charlottesville. “I tend to get the real hard cases,” Thomson told me recently. “A lot of the people I see have already tried multiple treatments. They arrive without much hope.” On one of the days I spent with Thomson earlier this winter, he checked his phone constantly for e-mail updates. A patient of his on “welfare watch” who was required to check in with him regularly had not done so, and Thomson was worried. “I’ve never gotten used to treating patients in mental pain,” he said. “Maybe it’s because every story is unique. You see one case of iron-deficiency anemia, you’ve seen them all. But the people who walk into my office are all hurting for a different reason.”

In the late 1990s, Thomson became interested in evolutionary psychology, which tries to explain the features of the human mind in terms of natural selection. The starting premise of the field is that the brain has a vast evolutionary history, and that this history shapes human nature. We are not a blank slate but a byproduct of imperfect adaptations, stuck with a mind that was designed to meet the needs of Pleistocene hunter-gatherers on the African savanna. While the specifics of evolutionary psychology remain controversial — it’s never easy proving theories about the distant past — its underlying assumption is largely accepted by mainstream scientists. There is no longer much debate over whether evolution sculptured the fleshy machine inside our head. Instead, researchers have moved on to new questions like when and how this sculpturing happened and which of our mental traits are adaptations and which are accidents.

In 2004, Thomson met Paul Andrews, an evolutionary psychologist at Virginia Commonwealth University, who had long been interested in the depression paradox — why a disorder that’s so costly is also so common. Andrews has long dark brown hair and an aquiline nose. Before he begins to talk, he often writes down an outline of his answer on scratch paper. “This is a very delicate subject,” he says. “I don’t want to say something reckless.”

Andrews and Thomson struck up an extended conversation on the evolutionary roots of depression. They began by focusing on the thought process that defines the disorder, which is known as rumination. (The verb is derived from the Latin word for “chewed over,” which describes the act of digestion in cattle, in which they swallow, regurgitate and then rechew their food.) In recent decades, psychiatry has come to see rumination as a dangerous mental habit, because it leads people to fixate on their flaws and problems, thus extending their negative moods. Consider “The Depressed Person,” a short story by David Foster Wallace, which chronicles a consciousness in the grip of the ruminative cycle. (Wallace struggled with severe depression for years before committing suicide in 2008.) The story is a long lament, a portrait of a mind hating itself, filled with sentences like this: “What terms might be used to describe such a solipsistic, self-consumed, bottomless emotional vacuum and sponge as she now appeared to herself to be?” The dark thoughts of “The Depressed Person” soon grow tedious and trying, but that’s precisely Wallace’s point. There is nothing profound about depressive rumination. There is just a recursive loop of woe.

The bleakness of this thought process helps explain why, according to the Yale psychologist Susan Nolen-Hoeksema, people with “ruminative tendencies” are more likely to become depressed. They’re also more likely to become unnerved by stressful events: for instance, Nolen-Hoeksema found that residents of San Francisco who self-identified as ruminators showed significantly more depressive symptoms after the 1989 Loma Prieta earthquake. And then there are the cognitive deficits. Because rumination hijacks the stream of consciousness — we become exquisitely attentive to our pain — numerous studies have found that depressed subjects struggle to think about anything else, just like Wallace’s character. The end result is poor performance on tests for memory and executive function, especially when the task involves lots of information. (These deficits disappear when test subjects are first distracted from their depression and thus better able to focus on the exercise.) Such research has reinforced the view that rumination is a useless kind of pessimism, a perfect waste of mental energy.

That, at least, was the scientific consensus when Andrews and Thomson began exploring the depression paradox. Their evolutionary perspective, however — they see the mind as a fine-tuned machine that is not prone to pointless programming bugs — led them to wonder if rumination had a purpose. They started with the observation that rumination was often a response to a specific psychological blow, like the death of a loved one or the loss of a job. (Darwin was plunged into a debilitating grief after his 10-year-old daughter, Annie, died following a bout of scarlet fever.) Although the D.S.M. manual, the diagnostic bible for psychiatrists, does not take such stressors into account when diagnosing depressive disorder — the exception is grief caused by bereavement, as long as the grief doesn’t last longer than two months — it’s clear that the problems of everyday life play a huge role in causing mental illness. “Of course, rumination is unpleasant,” Andrews says. “But it’s usually a response to something real, a real setback. It didn’t seem right that the brain would go haywire just when we need it most.”

Imagine, for instance, a depression triggered by a bitter divorce. The ruminations might take the form of regret (“I should have been a better spouse”), recurring counterfactuals (“What if I hadn’t had my affair?”) and anxiety about the future (“How will the kids deal with it? Can I afford my alimony payments?”). While such thoughts reinforce the depression — that’s why therapists try to stop the ruminative cycle — Andrews and Thomson wondered if they might also help people prepare for bachelorhood or allow people to learn from their mistakes. “I started thinking about how, even if you are depressed for a few months, the depression might be worth it if it helps you better understand social relationships,” Andrews says. “Maybe you realize you need to be less rigid or more loving. Those are insights that can come out of depression, and they can be very valuable.”

This radical idea — the scientists were suggesting that depressive disorder came with a net mental benefit — has a long intellectual history. Aristotle was there first, stating in the fourth century B.C. “that all men who have attained excellence in philosophy, in poetry, in art and in politics, even Socrates and Plato, had a melancholic habitus; indeed some suffered even from melancholic disease.” This belief was revived during the Renaissance, leading Milton to exclaim, in his poem “Il Penseroso”: “Hail divinest Melancholy/Whose saintly visage is too bright/To hit the sense of human sight.” The Romantic poets took the veneration of sadness to its logical extreme and described suffering as a prerequisite for the literary life. As Keats wrote, “Do you not see how necessary a World of Pains and troubles is to school an intelligence and make it a soul?”

But Andrews and Thomson weren’t interested in ancient aphorisms or poetic apologias. Their daunting challenge was to show how rumination might lead to improved outcomes, especially when it comes to solving life’s most difficult dilemmas. Their first speculations focused on the core features of depression, like the inability of depressed subjects to experience pleasure or their lack of interest in food, sex and social interactions. According to Andrews and Thomson, these awful symptoms came with a productive side effect, because they reduced the possibility of becoming distracted from the pressing problem.

The capacity for intense focus, they note, relies in large part on a brain area called the left ventrolateral prefrontal cortex (VLPFC), which is located a few inches behind the forehead. While this area has been associated with a wide variety of mental talents, like conceptual knowledge and verb conjugation, it seems to be especially important for maintaining attention. Experiments show that neurons in the VLPFC must fire continuously to keep us on task so that we don’t become sidetracked by irrelevant information. Furthermore, deficits in the VLPFC have been associated with attention-deficit disorder.

Several studies found an increase in brain activity (as measured indirectly by blood flow) in the VLPFC of depressed patients. Most recently, a paper to be published next month by neuroscientists in China found a spike in “functional connectivity” between the lateral prefrontal cortex and other parts of the brain in depressed patients, with more severe depressions leading to more prefrontal activity. One explanation for this finding is that the hyperactive VLPFC underlies rumination, allowing people to stay focused on their problem. (Andrews and Thomson argue that this relentless fixation also explains the cognitive deficits of depressed subjects, as they are too busy thinking about their real-life problems to bother with an artificial lab exercise; their VLPFC can’t be bothered to care.) Human attention is a scarce resource — the neural effects of depression make sure the resource is efficiently allocated.

But the reliance on the VLPFC doesn’t just lead us to fixate on our depressing situation; it also leads to an extremely analytical style of thinking. That’s because rumination is largely rooted in working memory, a kind of mental scratchpad that allows us to “work” with all the information stuck in consciousness. When people rely on working memory — and it doesn’t matter if they’re doing long division or contemplating a relationship gone wrong — they tend to think in a more deliberate fashion, breaking down their complex problems into their simpler parts.

The bad news is that this deliberate thought process is slow, tiresome and prone to distraction; the prefrontal cortex soon grows exhausted and gives out. Andrews and Thomson see depression as a way of bolstering our feeble analytical skills, making it easier to pay continuous attention to a difficult dilemma. The downcast mood and activation of the VLPFC are part of a “coordinated system” that, Andrews and Thomson say, exists “for the specific purpose of effectively analyzing the complex life problem that triggered the depression.” If depression didn’t exist — if we didn’t react to stress and trauma with endless ruminations — then we would be less likely to solve our predicaments. Wisdom isn’t cheap, and we pay for it with pain.

Consider a young professor on tenure track who was treated by Thomson. The patient was having difficulties with his academic department. “This guy was used to success coming easy, but now it wasn’t,” Thomson says. “I made it clear that I thought he’d need some time to figure out his next step. His problem was like a splinter, and the pain wouldn’t go away until the splinter was removed.” Should the patient leave the department? Should he leave academia? Or should he try to resolve the disagreement? Over the next several weeks, Thomson helped the patient analyze his situation and carefully think through the alternatives. “We took it one variable at a time,” Thomson says. “And it eventually became clear to him that the departmental issues couldn’t be fixed. He needed to leave. Once he came to that conclusion, he started feeling better.”

The publication of Andrews and Thomson’s 36,000-word paper in the July 2009 issue of Psychological Review had a polarizing effect on the field. While some researchers, like Jerome Wakefield, a professor at New York University who specializes in the conceptual foundations of clinical theory, greeted the paper as “an extremely important first step toward the re-evaluation of depression,” other psychiatrists regarded it as little more than irresponsible speculation, a justification for human suffering. Peter Kramer, a professor of psychiatry and human behavior at Brown University, describes the paper as “a ladder with a series of weak rungs.” Kramer has long defended the use of antidepressants — his landmark work, “Listening to Prozac,” chronicled the profound improvements of patients taking the drugs — and criticized those who romanticized depression, which he compares to the glamorization of tuberculosis in the late 19th century. In a series of e-mail messages to me, Kramer suggested that Andrews and Thomson neglect the variants of depression that don’t fit their evolutionary theory. “This study says nothing about chronic depression and the sort of self-hating, paralyzing, hopeless, circular rumination it inspires,” Kramer wrote. And what about post-stroke depression? Late-life depression? Extreme depressive condition? Kramer argues that there’s a clear category difference between a healthy response to social stressors and the response of people with depressive disorder. “Depression is not really like sadness,” Kramer has written. “It’s more an oppressive flattening of feeling.”

Even scientists who are sympathetic to what Andrews and Thomson call the “analytic-rumination hypothesis” remain critical of its details. Ed Hagen, an anthropologist at Washington State University who is working on a book with Andrews, says that while the analytic-rumination hypothesis has persuaded him that some depressive symptoms might improve problem-solving skills, he remains unconvinced that it is a sufficient explanation for depression. “Individuals with major depression often don’t groom, bathe and sometimes don’t even use the toilet,” Hagen says. They also significantly “reduce investment in child care,” which could have detrimental effects on the survival of offspring. The steep fitness costs of these behaviors, Hagen says, would not be offset by “more uninterrupted time to think.”

Other scientists, including Randolph Nesse at the University of Michigan, say that complex psychiatric disorders like depression rarely have simple evolutionary explanations. In fact, the analytic-rumination hypothesis is merely the latest attempt to explain the prevalence of depression. There is, for example, the “plea for help” theory, which suggests that depression is a way of eliciting assistance from loved ones. There’s also the “signal of defeat” hypothesis, which argues that feelings of despair after a loss in social status help prevent unnecessary attacks; we’re too busy sulking to fight back. And then there’s “depressive realism”: several studies have found that people with depression have a more accurate view of reality and are better at predicting future outcomes. While each of these speculations has scientific support, none are sufficient to explain an illness that afflicts so many people. The moral, Nesse says, is that sadness, like happiness, has many functions.

Although Nesse says he admires the analytic-rumination hypothesis, he adds that it fails to capture the heterogeneity of depressive disorder. Andrews and Thomson compare depression to a fever helping to fight off infection, but Nesse says a more accurate metaphor is chronic pain, which can arise for innumerable reasons. “Sometimes, the pain is going to have an organic source,” he says. “Maybe you’ve slipped a disc or pinched a nerve, in which case you’ve got to solve that underlying problem. But much of the time there is no origin for the pain. The pain itself is the dysfunction.”

Andrews and Thomson respond to such criticisms by acknowledging that depression is a vast continuum, a catch-all term for a spectrum of symptoms. While the analytic-rumination hypothesis might explain those patients reacting to an “acute stressor,” it can’t account for those whose suffering has no discernible cause or whose sadness refuses to lift for years at a time. “To say that depression can be useful doesn’t mean it’s always going to be useful,” Thomson says. “Sometimes, the symptoms can spiral out of control. The problem, though, is that as a society, we’ve come to see depression as something that must always be avoided or medicated away. We’ve been so eager to remove the stigma from depression that we’ve ended up stigmatizing sadness.”

For Thomson, this new theory of depression has directly affected his medical practice. “That’s the litmus test for me,” he says. “Do these ideas help me treat my patients better?” In recent years, Thomson has cut back on antidepressant prescriptions, because, he says, he now believes that the drugs can sometimes interfere with genuine recovery, making it harder for people to resolve their social dilemmas. “I remember one patient who came in and said she needed to reduce her dosage,” he says. “I asked her if the antidepressants were working, and she said something I’ll never forget. ‘Yes, they’re working great,’ she told me. ‘I feel so much better. But I’m still married to the same alcoholic son of a bitch. It’s just now he’s tolerable.’ ”

The point is the woman was depressed for a reason; her pain was about something. While the drugs made her feel better, no real progress was ever made. Thomson’s skepticism about antidepressants is bolstered by recent studies questioning their benefits, at least for patients with moderate depression. Consider a 2005 paper led by Steven Hollon, a psychologist at Vanderbilt University: he found that people on antidepressants had a 76 percent chance of relapse within a year when the drugs were discontinued. In contrast, patients given a form of cognitive talk therapy had a relapse rate of 31 percent. And Hollon’s data aren’t unusual: several studies found that patients treated with medication were approximately twice as likely to relapse as patients treated with cognitive behavior therapy. “The high relapse rate suggests that the drugs aren’t really solving anything,” Thomson says. “In fact, they seem to be interfering with the solution, so that patients are discouraged from dealing with their problems. We end up having to keep people on the drugs forever. It was as if these people have a bodily infection, and modern psychiatry is just treating their fever.”

Thomson describes a college student who was referred to his practice. “It was clear that this patient was in a lot of pain,” Thomson says. “He couldn’t sleep, couldn’t study. He had some family issues” — his parents were recently divorced — “and his father was exerting a tremendous amount of pressure on him to go to graduate school. Because he’s got a family history of depression, the standard of care would be to put him on drugs right away. And a few years ago, that’s what I would have done.”

Instead, Thomson was determined to help the student solve his problem. “What you’re trying to do is speed along the rumination process,” Thomson says. “Once you show people the dilemma they need to solve, they almost always start feeling better.” He cites as evidence a recent study that found “expressive writing” — asking depressed subjects to write essays about their feelings — led to significantly shorter depressive episodes. The reason, Thomson suggests, is that writing is a form of thinking, which enhances our natural problem-solving abilities. “This doesn’t mean there’s some miracle cure,” he says. “In most cases, the recovery period is going to be long and difficult. And that’s what I told this young student. I said: ‘I know you’re hurting. I know these problems seem impossible. But they’re not. And I can help you solve them.’ ”

IT’S TOO SOON to judge the analytic-rumination hypothesis. Nobody knows if depression is an adaptation or if Andrews and Thomson have merely spun another “Just So” story, a clever evolutionary tale that lacks direct evidence. Nevertheless, their speculation is part of a larger scientific re-evaluation of negative moods, which have long been seen as emotional states to avoid. The dismissal of sadness and its synonyms is perhaps best exemplified by the rise of positive psychology, a scientific field devoted to the pursuit of happiness. In recent years, a number of positive psychologists have written popular self-help books, like “The How of Happiness” and “Authentic Happiness,” that try to outline the scientific principles behind “lasting fulfillment” and “getting the life we want.”

The new research on negative moods, however, suggests that sadness comes with its own set of benefits and that even our most unpleasant feelings serve an important purpose. Joe Forgas, a social psychologist at the University of New South Wales in Australia, has repeatedly demonstrated in experiments that negative moods lead to better decisions in complex situations. The reason, Forgas suggests, is rooted in the intertwined nature of mood and cognition: sadness promotes “information-processing strategies best suited to dealing with more-demanding situations.” This helps explain why test subjects who are melancholy — Forgas induces the mood with a short film about death and cancer — are better at judging the accuracy of rumors and recalling past events; they’re also much less likely to stereotype strangers.

Last year Forgas ventured beyond the lab and began conducting studies in a small stationery store in suburban Sydney, Australia. The experiment itself was simple: Forgas placed a variety of trinkets, like toy soldiers, plastic animals and miniature cars, near the checkout counter. As shoppers exited, Forgas tested their memory, asking them to list as many of the items as possible. To control for the effect of mood, Forgas conducted the survey on gray, rainy days — he accentuated the weather by playing Verdi’s “Requiem” — and on sunny days, using a soundtrack of Gilbert and Sullivan. The results were clear: shoppers in the “low mood” condition remembered nearly four times as many of the trinkets. The wet weather made them sad, and their sadness made them more aware and attentive.

The enhancement of these mental skills might also explain the striking correlation between creative production and depressive disorders. In a survey led by the neuroscientist Nancy Andreasen, 30 writers from the Iowa Writers’ Workshop were interviewed about their mental history. Eighty percent of the writers met the formal diagnostic criteria for some form of depression. A similar theme emerged from biographical studies of British writers and artists by Kay Redfield Jamison, a professor of psychiatry at Johns Hopkins, who found that successful individuals were eight times as likely as people in the general population to suffer from major depressive illness.

Why is mental illness so closely associated with creativity? Andreasen argues that depression is intertwined with a “cognitive style” that makes people more likely to produce successful works of art. In the creative process, Andreasen says, “one of the most important qualities is persistence.” Based on the Iowa sample, Andreasen found that “successful writers are like prizefighters who keep on getting hit but won’t go down. They’ll stick with it until it’s right.” While Andreasen acknowledges the burden of mental illness — she quotes Robert Lowell on depression not being a “gift of the Muse” and describes his reliance on lithium to escape the pain — she argues that many forms of creativity benefit from the relentless focus it makes possible. “Unfortunately, this type of thinking is often inseparable from the suffering,” she says. “If you’re at the cutting edge, then you’re going to bleed.”

And then there’s the virtue of self-loathing, which is one of the symptoms of depression. When people are stuck in the ruminative spiral, their achievements become invisible; the mind is only interested in what has gone wrong. While this condition is typically linked to withdrawal and silence — people become unwilling to communicate — there’s some suggestive evidence that states of unhappiness can actually improve our expressive abilities. Forgas said he has found that sadness correlates with clearer and more compelling sentences, and that negative moods “promote a more concrete, accommodative and ultimately more successful communication style.” Because we’re more critical of what we’re writing, we produce more refined prose, the sentences polished by our angst. As Roland Barthes observed, “A creative writer is one for whom writing is a problem.”

This line of research led Andrews to conduct his own experiment, as he sought to better understand the link between negative mood and improved analytical abilities. He gave 115 undergraduates an abstract-reasoning test known as Raven’s Progressive Matrices, which requires subjects to identify a missing segment in a larger pattern. (Performance on the task strongly predicts general intelligence.) The first thing Andrews found was that nondepressed students showed an increase in “depressed affect” after taking the test. In other words, the mere presence of a challenging problem — even an abstract puzzle — induced a kind of attentive trance, which led to feelings of sadness. It doesn’t matter if we’re working on a mathematical equation or working through a broken heart: the anatomy of focus is inseparable from the anatomy of melancholy. This suggests that depressive disorder is an extreme form of an ordinary thought process, part of the dismal machinery that draws us toward our problems, like a magnet to metal.

But is that closeness effective? Does the despondency help us solve anything? Andrews found a significant correlation between depressed affect and individual performance on the intelligence test, at least once the subjects were distracted from their pain: lower moods were associated with higher scores. “The results were clear,” Andrews says. “Depressed affect made people think better.” The challenge, of course, is persuading people to accept their misery, to embrace the tonic of despair. To say that depression has a purpose or that sadness makes us smarter says nothing about its awfulness. A fever, after all, might have benefits, but we still take pills to make it go away. This is the paradox of evolution: even if our pain is useful, the urge to escape from the pain remains the most powerful instinct of all.

Correction Appended: An earlier version of this article incorrectly referred to the University of New South Wales as the University of South Wales.